Repository navigation
Complete native Volume transfers for ExecContext - #560
aakash-saravanan-db wants to merge 20 commits into
Conversation
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
|
Go integration tests triggered ( |
There was a problem hiding this comment.
Verdict: 1 Low · 1 Nit
Mostly sound, with two low-severity items. The native staging dispatch, cancellation watcher, and defer teardown order look correct. REMOVE not-found=success matches Thrift's 404 handling. The kernel-side behavior depends on the pinned kernel revision, which isn't in this checkout, so I couldn't check it here. Nit: the error strings "Volume transfer: ..." and "Volume operation requires..." start with a capital letter, which is against Go convention (staticcheck ST1005).
Other findings
- ⚪ Nit —
nativeStagingStubhas anerrfield, but no table case ever sets it. The dispatch path that wraps aCompleteStagingfailure inNewDriverError(..., "error performing staging operation", err)is never tested. No test checks that the returned error is non-nil in that case, or that the cause can still be reached witherrors.Is/errors.As(e.g.context.Canceled,*os.PathError). That cause chain is the point of the newKernelError.Unwrapand the ctx-wrapping inCompleteStaging. A case witherr: context.Canceledthat assertserrors.Is(err, context.Canceled)would cover it.
| C.kernel_result_stream_canceller_cancel(canceller) | ||
| case <-done: | ||
| } | ||
| }() |
There was a problem hiding this comment.
🔵 Low — CompleteStaging returns toStatementError(err) for failures from get_result_stream, canceller_new, and complete_volume_operation, but it never calls o.backend.evictIfSessionFatal(err). Both Results (L373-382) and execute make that call on the same kinds of kernel errors. Suppose the token expires or the session is dropped while a Volume transfer completes. The error still reaches the caller, but the conn stays marked valid, so the pool keeps handing out a dead connection. The non-staging read path doesn't have this problem. IsStaging's error path has the same gap. Calling o.backend.evictIfSessionFatal(err) before returning in these branches would match Results.
There was a problem hiding this comment.
I checked the reachable error sources before adding eviction. kernel_executed_statement_get_result_stream only validates/takes the existing synchronous stream (src/c_abi/statement.rs:733–772); marker and detached-canceller creation are local ABI operations. They cannot observe an expired token or dropped remote session.
Completion can encounter both result transport and presigned-storage failures. Their shared kernel error codes do not establish that the SQL session is dead; applying evictIfSessionFatal indiscriminately could discard a healthy SQL connection on a storage authentication/permission failure. I have not found a demonstrated session-origin failure requiring a change here. No message-prefix classifier or new error-origin framework was added for this hypothetical case.
The native source pin is updated to 22f96fac for the required shared retries. Native-enabled package tests and the final named two-backend Volume workflow pass with exact bytes/deletion proof. A concrete session-origin reproduction would be useful evidence to reassess selective eviction.
There was a problem hiding this comment.
You're right about get_result_stream and the canceller: they're local ABI operations, so they can't report a dead session, and that part of my finding was wrong. The complete_volume_operation branch still looks inconsistent to me, though. isSessionFatal only matches Unauthenticated, Unavailable and NetworkError. Results/rows.go already call evictIfSessionFatal on those codes even when the error comes from CloudFetch downloads to presigned storage, which carry the same "storage, not session" ambiguity you describe. So the trade-off you're avoiding here is one the read path already makes. Accepting it in one place and not the other is reasonable, but it's a call for a maintainer. The worst case is a dead conn going back to the pool after a Volume completion fails with an auth or transport error. That's low severity, so I'll leave the thread open for a human to decide instead of pushing further.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
|
Integration test approval reset. New commits were pushed to this PR. Label(s) A maintainer must re-review and re-add a label to preview tests again. (The real gate runs in the merge queue.) Latest commit: 26f142a |
There was a problem hiding this comment.
Verdict: 1 Medium · 2 Low
The dispatch is mostly sound, with one medium lint concern in the new kernel CompleteStaging path. ExecContext now routes kernel Volume operations to native completion, and Thrift keeps its existing row-based path. The rest are low-severity issues: missing session-fatal eviction and thin test coverage.
| return toStatementError(err) | ||
| } | ||
| defer C.kernel_result_stream_close(stream) | ||
| var canceller *C.kernel_result_stream_canceller_t |
There was a problem hiding this comment.
🔵 Low — CompleteStaging converts kernel failures with toStatementError but never calls o.backend.evictIfSessionFatal(err). That applies to get_result_stream, canceller_new and complete_volume_operation. Every other kernel path that runs after execute does call it, including Results on the same get_result_stream call and every failure branch in execute. A completion that runs long enough for the token to expire, or for the session to be dropped, will return the error but leave the conn marked valid in the pool. The next statement on that conn then fails as well. Call o.backend.evictIfSessionFatal(err) on these failure branches, the same way Results does.
|
Go integration tests triggered ( |
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
|
Integration test approval reset. New commits were pushed to this PR. Label(s) A maintainer must re-review and re-add a label to preview tests again. (The real gate runs in the merge queue.) Latest commit: f0c6f41 |
There was a problem hiding this comment.
Verdict: 1 Medium · 1 Nit
Mostly sound. One medium issue: when native staging completion fails with a session-fatal error, the session is not marked dead, unlike every other kernel result path. Kernel-side test coverage for the new paths is also thin. Nit: the new CHANGELOG entry is followed by a stray blank line inside the bullet list.
Other findings
- ⚪ Nit — None of the new kernel-side behaviour has unit coverage. Untested paths:
kernelOp.IsStagingwith a nil exec.- The
CompleteStagingoutcome mapping (outcome != 2returns an error). - Wrapping of the cancelled-context error.
- The new
KernelError.Unwrap/causepath, whereerrors.Is(err, fs.ErrNotExist)should work through a*KernelErrorcarrying anos.PathError.
The only new test (TestConnNativeStagingDispatch) uses a Go stub at the conn layer. The removed TestExecuteRejectsStaging has no replacement, and driver_e2e_test.go has no kernel staging test. Unwrap could be pinned in the non-cgo errors_classify_test.go. kernel_test.go could cover IsStaging with a nil exec and the outcome mapping.
|
Go integration tests triggered ( |
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
|
Integration test approval reset. New commits were pushed to this PR. Label(s) A maintainer must re-review and re-add a label to preview tests again. (The real gate runs in the merge queue.) Latest commit: 3629843 |
There was a problem hiding this comment.
Verdict: 1 Medium · 2 Low
The PR looks correct overall. There is one medium concern: CompleteStaging skips evicting a dead session when a session-fatal error races context cancellation, which execute() deliberately avoids. There are also two low gaps: no eviction on stream-acquisition failure, and no test for the cancellation path.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
Verdict: 2 Low
The dispatch and cancellation design look sound, with two low-severity concerns: two early-failure paths in CompleteStaging skip session-fatal eviction, and the native path drops two telemetry tags. As the PR description already says, the pinned KERNEL_REV and header point at an unmerged kernel commit and need exports that the published binding modules don't have, so databricks_kernel-tagged builds against published bindings will fail to link until those modules and the module/sum updates ship.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Medium · 1 Low · 1 Nit
Mostly solid, with one medium gap: the session-fatal eviction this PR adds is skipped on the early get_result_stream failure path in CompleteStaging. Also worth noting for release: KERNEL_REV pins an unmerged kernel PR commit, and the new C symbols need a kernel release later than 1.4.0 before this can ship. The cancellation test relies on the exact number of ctx.Err() calls (errCalls == 2), which is fragile (nit).
Other findings
- 🔵 Low — Removing the
isStagingStatementpre-check means staging detection now relies only on the server manifest'sis_volume_operationflag, reported throughkernel_executed_statement_is_volume_operation. If a warehouse or result shape leaves that flag out (or the kernel misses it),IsStagingreturns false, andExecContexton a PUT/GET reports success with 0 rows while no file moves. That is the silent-data-loss case the removed guard existed to block. Consider keeping the SQL-text detector as a fallback: if the text looks like PUT/GET/REMOVE but the operation is not marked, return an error instead of success. Alternatively, record that the server is guaranteed to always set the flag. - ⚪ Nit — The native branch returns before the
telemetryUpdatecallback is wired, so kernel Volume transfers never setTagChunkCount/TagBytesDownloaded. The Thrift staging path does set them. This is a minor telemetry gap compared with Thrift. If the C ABI can't report byte counts, a short comment noting the omission would help.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low · 2 Nit
Mostly sound, with 3 low concerns: the session-eviction policy is applied inconsistently on the early error paths of CompleteStaging, one error message is misleading for status 0, and the new cancellation watcher has no test. Follow-up outside this PR: per the description, the published 1.4.0 binding archives lack the new kernel_get_last_error_* and volume exports, so a -tags databricks_kernel build against the published bindings will fail to link until a matching bindings release ships.
Other findings
- ⚪ Nit — The
outcome != 2branch handles two different cases: status 0 ("ordinary result": the stream turned out not to be a Volume operation) and status 1 ("input bytes required"). Both get the same error,volume operation requires a local file instruction, and that message only makes sense for status 1. A status-0 result afterIsStagingreturned true points to a kernel or server inconsistency, and this message would send whoever debugs it in the wrong direction. Neither case is tested.staging_kernel_test.goonly covers error returns from completion, andTestConnNativeStagingDispatchstubs outCompleteStaging. Consider using a distinct message for each status (including the numeric status helps) and adding a test for it. - ⚪ Nit — The
*Cancelledsubtests do not exercise the cancellation path this PR adds.cancelOnCompletionErrorContextcancels on the secondErr()call, which comes afterkernel_result_stream_complete_volume_operationhas already returned an error. So the watcher goroutine never callskernel_result_stream_canceller_cancelwhile a transfer is running. These cases only check how the error is wrapped after the fact. They also depend on how many timesCompleteStaginghappens to callctx.Err(), so adding or removing one check would silently change what they test. A test that cancels while the storage request is in flight (for example, a handler that blocks until the ctx is cancelled) would cover the documented promise that "context cancellation" interrupts the transfer.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low · 1 Nit
Looks good overall, with 2 low concerns: some kernel CompleteStaging setup failures skip the session-eviction policy, and nothing tests that cancellation actually interrupts an in-flight transfer. Follow-up (not caused by this PR): the native path doesn't emit the chunk-count/bytes telemetry tags that the Thrift staging path does.
Other findings
- ⚪ Nit — Nothing tests the result-stream canceller watcher (
case <-ctx.Done(): C.kernel_result_stream_canceller_cancel(canceller)). The two*Cancelledsubtests instaging_kernel_test.gocancel the context from insideErr()only after native completion has already returned. They exercise the error-wrapping branch, not an interrupt of an in-flight transfer. If the watcher were dropped or never fired, a cancelled ExecContext would block until the storage transfer finished and every test would still pass. Consider adding a case where the/objecthandler blocks until the request context is cancelled, then assert thatExecContextreturnscontext.Canceledpromptly.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good overall, with one low concern. Native Volume completion now goes through CompleteStaging. The canceller and deferred cleanup run in the correct LIFO order, and storage failures leave the session valid. When the instruction stream fails to open, CompleteStaging doesn't apply the session-eviction policy that Results applies to the same call. Nit: the VOLUME_STORAGE_* error-code strings are inline literals and could be named constants.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Medium
Looks mostly correct, with 1 medium issue: the new CompleteStaging doesn't evict a dead session when getting the result stream or the canceller fails. Otherwise the dispatch, cancellation wiring, defer ordering, and the storage-vs-instruction eviction split look right. Follow-up: the native path no longer emits the chunk-count and bytes-downloaded telemetry tags that the Thrift staging path records.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
The change looks sound, with one low-severity concern: the kernel example still describes staging as rejected on the kernel backend. CompleteStaging cleans up in a safe order (stops the watcher goroutine, then frees the canceller, then closes the stream). It only evicts the session for non-storage failures, and it never returns ErrBadConn. Nits: the cancellation test depends on the exact number of ctx.Err() calls (errCalls==2), so a harmless refactor could break it. The native completion path also skips the chunk-count/bytes telemetry tags that the Thrift staging path records.
Other findings
- 🔵 Low — This PR removes the up-front staging rejection from
Execute, butexamples/kernel/main.gostill says staging is "rejected at execute". It also runs a realPUT '/tmp/local.csv' INTO '/Volumes/main/default/vol/f.csv' OVERWRITEand expectsErrNotSupportedByKernel.
After this change, that statement goes to the server. With no staging context set, it fails with "staging ctx must be provided." instead of the documented rejection. If someone copies the example and adds staging roots, the PUT really runs and overwrites the Volume file. The example and its comment should be updated or the staging block removed, along with the doc.go/CHANGELOG edits.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
|
The obsolete staging-rejection example is removed in 6218f29. The example retains backend selection, connection handling and SELECT; it no longer submits the hard-coded PUT. The cancellation fault-injection test's dependence on the |
There was a problem hiding this comment.
Verdict: 1 Low
Looks mostly good, with one low concern. Native completion goes through ExecContext cleanly, and the storage-vs-instruction eviction split is tested against the real native library, including races with cancellation. Possible follow-up: the native staging path no longer records chunk/byte telemetry tags the way the Results-based path did.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Medium
Looks good overall, with one medium gap. On two early setup failures in CompleteStaging, the session-fatal eviction that Results() applies is skipped. Not blocking but worth a follow-up: the native path drops the chunk/bytes telemetry tags that the Thrift staging path sets, and no tagged test covers a successful native completion or the outcome != 2 branch.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good overall. I found one low-severity gap: the early error paths in CompleteStaging skip the session-eviction policy. The dispatch, cancellation teardown order, ErrBadConn avoidance and REMOVE-404 parity with Thrift all check out. A possible follow-up: the native path records no chunk/bytes telemetry tags, while the Thrift staging path does.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good overall, with one low concern: CompleteStaging doesn't apply the fatal-session eviction when it fails to get the result stream or create the canceller. The dispatch in execStagingOperation, the cancellation wrapping and the storage-vs-instruction eviction split look correct. The new error-code values (VOLUME_STORAGE_HTTP_ERROR and the others) are typed in as string literals; named constants would be a nice touch (nit).
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good overall, with one low-severity gap. When opening the result stream or its canceller fails inside CompleteStaging, the session is not evicted, unlike in Results. As a nit, the native path no longer adds the chunk-count/bytes telemetry tags that the Thrift staging path adds, which could be follow-up work.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Medium · 1 Low
Mostly sound, with one medium test-coverage gap and one low inconsistency in session eviction. The dispatch, cancellation watcher, defer ordering and storage/instruction error split look correct. As follow-up: the native path does not record the chunk-count and bytes-downloaded telemetry tags that the Thrift staging path reports, and the storage exemption depends on two hard-coded kernel error-code strings.
| return c.Context.Err() | ||
| } | ||
|
|
||
| func TestKernelVolumeFailureOriginControlsSessionValidity(t *testing.T) { |
There was a problem hiding this comment.
🟡 Medium — The only in-repo test of the native path covers failures: instruction 401, storage 401, a short body, and the two cancelled variants. Nothing tests the main feature the CHANGELOG and doc.go describe:
- a GET or PUT through
ExecContextthat succeeds and writes or uploads the bytes, then reports zero rows affected; - a REMOVE that succeeds, including a 404 treated as success because of
remove_not_found_is_success=true; - the new rule that rejects a symlink whose resolved target leaves the staging roots.
TestConnNativeStagingDispatch stubs CompleteStaging, so it never reaches kernelOp.CompleteStaging or the C ABI. The httptest harness already in this file can do this. Add a success mode that checks the destination's new contents and IsValid()==true. Add a mode where localFile is a symlink inside directory that points outside it, and assert an error with no storage request. Without these, a kernel bump that breaks the success outcome (outcome != 2) or the symlink check would pass CI.
There was a problem hiding this comment.
The Go-local native test does cover failure/cancellation rather than successful transfer or escaped symlinks; expanding it would be useful optional regression coverage. Required behavior is covered by distinct groups: current shared-core tests cover resolved-root rejection and permissive REMOVE404; the C ABI maps Completed directly to outcome2; and the actual source-built Go ExecContext lifecycle verifies initial/changed PUT, GET bytes and REMOVE against Thrift with one match and zero differences. That retained comparison used Go513f8e73 and native archivecec365e6; the current executable Go adapter is byte-identical, with only its source pin changed. The retained native predates the later common-core reader/empty-batch corrections, which have fresh shared-core checks. The live run does not prove Go-local symlink or REMOVE404 cases. A duplicate connector matrix is not needed for this correction; package/install and named scheduled execution gates remain open.
There was a problem hiding this comment.
Thanks, that's a fair summary of where coverage lives. Shared-core tests and the outcome-2 mapping in the C ABI do cover the logic itself. But those tests run in the kernel's CI, and the source-built live comparison was a one-off check against a pinned archive, not a gate in this repo. You also note that it doesn't cover the Go-local symlink or REMOVE-404 cases. So the risk I raised is still there. If a future kernel bump or adapter edit breaks how kernelOp.CompleteStaging handles a successful outcome, CI here would still pass. One success mode in the existing httptest harness would close most of that gap: a GET that checks the destination bytes, IsValid()==true and zero rows affected. A second mode would cover the symlink: a symlink that escapes the staging roots, asserting an error and no storage request. Leaving this open so a human can decide whether to add those tests now or track them as follow-up.
Feature or gap
Kernel-backed Go connections need to complete Volume operations with existing staging authorization and error handling.
What this PR addresses
Delegate transfers to shared native completion and preserve cancellation causes. Apply the existing fatal-session policy to SQL instruction errors while preserving sessions on storage HTTP/body failures, including errors racing context cancellation, without returning ErrBadConn or replaying the statement.
Dependencies
Builds select databricks/databricks-sql-kernel#465 at d9fcf834d36dc7f3de1621225db716200151fa0f and its matching C header. Merge and release require compatible binding versions, updated modules/sums, and clean installed-package validation. Published 1.4.0 archives lack the new exports.
Tests and validation
Retained validation against native archive
cec365e6passes 48 focused native-enabled checks, including failure-origin/cancellation and closed-connection/telemetry controls. The retained source-built named file lifecycle matches Thrift on DBSQL CURRENT: one matching workflow and zero differences, with initial bytes, changed overwrite, storage-origin removal and owned cleanup verified. The recorded executable links the audited kernel archive; the instruction-reader error-telemetry path is verified through focused shared-core tests. Retained default/full-suite checks, the invalidation-ordering negative control and performance comparisons keep their recorded build qualifications. Clean installed-package and scheduled activation remain open.